Performance of networks of artificial neurons: the role of clustering 
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The performance of the Hopfield neural network model is numerically studied on various complex 
networks, such as the Watts-Strogatz network, the Barabasi- Albert network, and the neuronal 
network of the C. elegans. Through the use of a systematic way of controlling the clustering 
coefficient, with the degree of each neuron kept unchanged, we find that the networks with the lower 
clustering exhibit much better performance. The results are discussed in the practical viewpoint of 
application, and the biological implications are also suggested. 

PACS numbers: 84.35.+i, 89.75.Hc, 87.17.-d 



Recently, researches related with complex networks 
have been broadening territory beyond individual disci- 
plines. Startingfrom pioneering works of modeling com- 
plex networks 0, Q , the essential network concepts have 
been successfully applied to various systems covering bi- 
ological networks, food webs, Internet, e-mail network, 
and so on Q. In parallel to the studies of structural 
properties of complex networks, there have also been 
strong interest in dynamic systems defined on networks. 
For examples, non-Hamiltonian dynamic models such as 
epidemic spread cascading failures synchroniza- 
tion the sandpile 0, and the prisoner's dilemma 
game |9| , as well as equilibrium statistical physics models 
such as the Ising ^} and the XY models have been 
investigated. 

The network of neurons in biological organisms also 
takes the form of complex networks I31IT^ .IT3 . IT^ |. While 
the C. elegans pi El Ell an d the in vitro [lj neuronal net- 
works have the high level of clustering, the small charac- 
teristic path length, and the degree distribution far from 
scale-free in common, the functional network of human 
brains 0] has recently been revealed to be scale-free. 
Motivated by the neurons connected by synaptic cou- 
plings in biological organisms, simple mathematical mod- 
els of artificial neurons have been suggested One of 
the practical applications of such an artificial neural net- 
work can be found in the Hopfield model |17| . which is 
used frequently in the pattern recognition. Very recently, 
there have been studies of the Hopfield model of neurons 
put on the structure of complex networks 0, , with 
major focus on how the topology, the degree distribu- 
tion in particular, of a network affects the computational 
performance of the Hopfield model. Also in the neu- 
roscience, recent investigations have revealed the close 
interrelationship between the brain activity and the un- 
derlying neuroanatomy |l3j . 

Table [I] summarizes the clustering coefficients (see 
Ref. [l| for the definition) and the performance of the 
Hopfield model on various network structures [2(j (see 
below for details). The difference in the performance, 



measured by the overlap m between the neuron state 
and the stored pattern, has been previously attributed to 
the distinct network topology, or the degree distribution 
more specifically [l^. Examining Table |U in more detail, 
one can easily recognize the systematic dependence of the 
network performance on the clustering property: As the 
clustering becomes weaker, the performance is enhanced 
monotonically. The performance detected by the ratio 
R (the last column in Table. [I] see Ref. EH) of the num- 
ber of correctly recalled bits to the number of synaptic 
couplings again shows the same behavior. 

We in this work study the performance of the Hop- 
field model on various network structures with focus on 
the role of the clustering. We extend the edge exchange 
method in Ref. and suggest a systematic way to con- 
trol the clustering coefficient of a given network without 
changing the degree of each vertex. A set of networks 
with the identical degree distribution but with various 
clustering coefficients are generated and then used as the 
underlying network structures for the numerical simula- 
tions of the Hopfield model. Clearly revealed is that the 
computational performance depends much more strongly 
on the clustering property than on the degree distribu- 
tion. 

We begin with a brief review of the edge exchange 



TABLE I: Performance of Hopfield model on networks. All 
networks are of the same size N = 280 and the average de- 
gree (k) ?s 14. The neuronal network of the worm C. elegans, 
the Barabasi- Albert (BA) network, and the Watts-Strogatz 
(WS) networks at the rewiring probabilities P — 0.0, 0.1, and 
1.0 are used. Both the overlap m and the input-output ra- 
tio R, measuring the performance of the network, increase as 
the clustering coefficient 7 is decreased. The maximum de- 
gree fc max is also shown to give some rough estimation of the 
broadness of the degree distribution. 
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FIG. 1: Exchange of two edges: Two edges [A-B and C-D in 
(a)] are randomly picked and then rewired to have different 
end vertices [ A-D and C-B as in (b)]. The edge exchange 
keeps the degree of each vertex unchanged. 

method, recently suggested by Maslov and Sneppen in 
Ref. El Two edges, one connecting vertices A and B, 
and the other C and D, are randomly chosen [Fig.QIa)]. 
Each vertex changes its partner and the original edges 
A-B and C-D are altered to A-D and B-C as shown in 
Fig. ^b) . The important property of the above edge ex- 
change is that this process does not change the degree 
of each vertex. A blind repetitions of the above edge 
exchanges have been shown to destroy all degree-degree 
correlations [l6Ll22T| . 

Based on the above edge exchange method, we in the 
present paper introduce an additional acceptance stage: 
The edge exchange trial is accepted only when the new 
network configuration has higher (or lower) clustering co- 
efficient. This is identical to the standard Monte Carlo 
(MC) simulation at zero temperature (T) with the Hamil- 
tonian H set to H = - J2 V lv (or H = J2 V lv), where j v 
is the clustering coefficient of the vertex v [l|. In addi- 
tion, we also apply the standard MC scheme to the above 
Hamiltonian at finite temperatures. In either way, one 
can control the clustering coefficient of a given network 
with the degree of each vertex kept fixed. 

We use various network structures as underlying net- 
works of the Hopfield model. Since the real neuronal net- 
work of C. elegans has N = 280 vertices and the average 
degree (k) « 14, we generate various model networks of 
the size N — 280 such as Barabasi- Albert (BA) network 
with Mq = AI = 7 (starting from Mq vertices, one vertex 
with M edges is added at each step in the BA model 0), 
and the Watts- Strogatz (WS) networks with the connec- 
tion range r = 7 at the rewiring probabilities P = 
(corresponding to a regular local one-dimensional net- 
work), P = 0.1 (well inside in the small- world regime Q]), 
and P = 1.0 (corresponding to a fully random network). 
All the networks have different topologies in the sense 
that each has its own unique degree distribution. After 
the original networks are built, we decrease or increase 
the clustering coefficients through the use of the above 
zero-temperature MC method. Once the target value 
of the clustering coefficient is reached, the current net- 
work structure is frozen and then used as an underlying 
network structure for the Hopfield model. We also per- 
form MC at finite temperatures with the Hamiltonian 
H = — 7„ to generate networks at a given T for com- 



parisons. 

In the Hopfield model of a neural network ^j, a neu ~ 
ron at the site i can have two states, firing (cr.; = 1) or 
not firing (oi = —1). The neuron state Oi(i) at time t 
is determined from the configuration of other neurons at 
t — 1 according to 

cxiit) = sgn fa Atf^W* - !) j . (!) 

where Wy is the strength of the synaptic coupling be- 
tween neurons i and j, and Ay (= 0, 1) describes the con- 
nection structure of the neural network. For example, in 
the original version of the Hopfield neural network the 
couplings are of the mean-field type and thus Ay = 1 is 
taken for any pair (z, j) of two neurons. In the standard 
graph theory, Ay is simply the (zj)-component of the so- 
called adjacency matrix, i.e., Ay = 1 if two vertices i 
and j are connected while Ay = otherwise. For sim- 
plicity, we in this work consider only undirected networks 
and accordingly A is a symmetric matrix. Recently, Hop- 
field model on the asymmetric directed BA network has 
been studied 0|. According to Eq. the firing of 
neighbor neurons connected to the ith neuron via exci- 
tatory synaptic couplings (Wy > 0) leads to the firing of 
ith neuron at next time step, while inhibitory couplings 
(Wij < 0) inhibits the firing of ith neuron. 

For the task of the recognition of p stored patterns, 
the synaptic coupling strength Wy is usually given by 
the Hcbbian learning rule: 

Wtf = EW ( 2 ) 

where £f is the ith component of the /xth stored pattern 
vector and is quenched random variable taking values 1 
and —1 with equal probability. In this work, we present 
results only for p — 5 (we also tried p = 10 and 20 only 
to find insignificant qualitative differences). The initial 
neuron state configuration {ai(t = 0)} is produced from 
the one of the pattern vectors, say the ^th pattern, with 
20% error. In other words, cr,(0) = & for 80% of the 
neurons, while the other 20% has the reversed bit <7i(0) = 
As the neuron configuration evolves in time by 
Eq. the overlap m defined by 

i 

is measured as a function of time. The complete recog- 
nition of the pattern v gives m = 1, which corresponds 
to the null Hamming distance d(t) = {1/N) XX^M ~ 
£") 2 = 2 — 2m(t). We also measure the input-output 
ratio R of the correctly recalled pixels to the number 
of synaptic couplings, which is written as R = (1 + 
m)p/{k) [20L l2l| . The asynchronous dynamics, in which 
a neuron is chosen at random at each time step, is used 
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FIG. 2: Efficiency of the Hopfield neural network on the C. el- 
egans neuronal network, Barabasi-Albert (BA) network, and 
Watts-Strogatz (WS) network at the rewiring probabilities 
P — 0,0.1, and 1.0. The overlap m between the stored pat- 
tern and the neuron state measures the performance of each 
network and is plotted as a function of the clustering coeffi- 
cient 7 = (53 7„)/JV. The big filled circles are for the original 
networks before the edge exchange MC is applied (see text for 
details). It should be noted that at all points on each curve, 
not only the degree distribution but also the actual degree of 
each neuron is exactly identical. Performance of each network 
monotonically decreases as the clustering becomes stronger. 



throughout the present work. After sufficiently long runs 
of dynamic evolution up to t = 2000, where the one time 
unit corresponds to the one whole sweep of all neurons, 
the last 200 time steps are used to make the time average 
of m(t), and disorder averages over 1000 different pattern 
realizations are performed. 

Figure [3 is the main result of the present paper. For 
each network, we start from the original network struc- 
ture (denoted by the big filled circle on each curve), 
and increase (or decrease) the clustering coefficient by 
using the edge exchange zero-temperature MC method 
described above. Once the target value of the cluster- 
ing coefficient (7 = 0.0, 0.05, 0.10, 0.15, •■• ) is achieved 
the network structure is saved and used for the Hop- 
field model simulations. We repeat this procedure for 
5-10 times to make an average over network structures. 
We also present the result obtained for finite-temperature 
MC method in Fig.|3 We start from the original network 
at sufficiently high temperature T = 1 and use the stan- 
dard Metropolis algorithm applied for H = — J2 V lv- At 
each T, we equilibrate the network for 10 4 MC steps and 
perform the Hopfield model dynamic simulation. The 
temperature is then decreased slowly to make equilibra- 
tion at each T more efficient. It is to be noted that we 
have introduced the finite temperature only when we gen- 
erate networks by edge-exchange; the time evolution of 
neuron states in Eq. is still deterministic. 

The most important conclusion one can make out of 
Figs. 0and |3{c) is that the performance of the Hopfield 
model on various networks can be enhanced significantly 
if the clustering is weakened by the simple method of edge 
exchanges without altering the degree of each neuron. 
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FIG. 3: Networks are generated from the finite-temperature 
edge-exchange MC for the Hamiltonian H = — ^, v fv- (a) 
Clustering coefficient 7 versus the temperature T. (b) The 
overlap m versus T. From (a) and (b), m as a function of 7 
is obtained in (c). 



There are of course differences in performance among net- 
works at the same 7, e.g., for lower 7 WS networks are 
more efficient than the BA and the C. elegans, while the 
trend becomes opposite for higher 7, implying that the 
clustering coefficient is not the single parameter control- 
ling the performance. However, Figs.|21and|3Ic) strongly 
suggest that the efficiency of Hopfield neural network de- 
pends much more strongly on 7 than on the degree distri- 
bution as widely believed. Consequently, we suggest that 
the difference in the pattern recognition performance in 
Tablets strongly related with the clustering property of 
each network. The result of the importance of the clus- 
tering in the task of the pattern recognition by using the 
Hopfield model has also some practical importance: It 
provides a novel way to enhance the performance with- 
out adding more synaptic couplings 20] . For example, 
the simple ID Hopfield network can be made about 30% 
more efficient by exchanging pairs of edges. 

The WS networks at P = 0.0, 0.1, and 1.0 exhibit al- 
most the same performance for small clustering coeffi- 
cient. As the clustering becomes strong the WS network 
with P = 1.0 has better performance than the other 
smaller values of P [23|. However, the overall behavior 
does not show significant differences. On the other hand, 
it is interesting to note that the BA and the C. elegans 
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networks exhibit almost the same high performance at 
large 7, which appears to imply the importance of hub 
vertices (the maximum degree in the C.elegans network is 
77 while it is 67 for the BA network as shown in TablcQJ. 

In the viewpoint of biological evolution, it is not clear 
why the evolution chose the very structure of the neu- 
ronal network of C.elegans. One expects that the actual 
detailed structure of the neuronal network of C. elegans 
must have some advantage over other structures, which 
then leads to a very crude expectation that the advan- 
tage may be detected by the measurement of the perfor- 
mance of the Hopfield model, as has been investigated 
in this work. One somehow unlikely explanation is that 
the worm is still in the evolutionary process of optimiz- 
ing its neuronal connection. The other explanation can 
be the cost of the long-range synaptic couplings: The 
actual connection topology of C.elegans can be the best 
that the evolution can find if we consider the competi- 
tion between the performance and the (energy) cost to 
make long connections. This is somehow likely because 



the present work does not take the geometric constraint 
into account. Similar question on the competition be- 
tween the total axonal length (measuring energy cost for 
long-range connection) and the characteristic path length 
(measuring the efficiency of wiring structure) has been 
recently pursued in Ref. 14 and the optimization of the 
neuronal-component placement on geometry has been in- 
vestigated in Ref. yjj One can also think of other possi- 
bility that the better performance by removing cluster- 
ing has dark side effect, like vulnerability of performance 
under malfunctioning of neurons for example. In this 
respect, it may also draw some interest that the perfor- 
mance curve for C.elegans in Fig. [21 is the most flat one 
around the point for the actual original network. 
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Ajou University. Numerical works have been performed 
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